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Abstract 


Hysteresis  in  smart  materials  hinders  their  wider  applicability  in  actuators.  In  this  report  we 
investigate  control  of  smart  actuators  through  the  example  of  controlling  a  commercially  available 
magnetostrictive  actuator.  At  low  frequencies,  the  magnetostriction  can  be  related  to  the  bulk 
magnetization  through  a  square  law,  thus  control  of  the  magnetization  amounts  to  control  of  the 
magnetostriction.  The  model  we  use  is  the  low  dimensional  Jiles- Atherton  model  for  ferromagnetic 
hysteresis,  which  is  a  hybrid  system.  For  illustrative  purpose,  we  consider  an  infinite  horizon  control 
problem.  The  approach  we  take  features  dynamic  programming  and  Hamilton- Jacobi  equations. 
In  particular,  we  show  that  the  value  function  of  the  control  problem  satisfies  a  Hamilton- Jacobi- 
Bellman  equation  (HJB)  of  some  hybrid  form  in  the  viscosity  sense.  We  further  prove  uniqueness 
of  solutions  to  the  (HJB),  and  provide  a  numerical  scheme  to  approximate  the  solution  together 
with  a  suboptimal  controller  synthesis  method. 


1  Introduction 


Hysteresis  in  smart  materials,  e.g.,  magnetostrictives,  piezoceranrics,  and  shape  memory  alloys 
(SMAs),  hinders  the  wider  applicability  of  such  materials  in  actuators.  Hysteresis  models  can  be 
classified  into  physics  based  models  and  phenomenological  models.  An  example  of  physics  based 
model  is  the  Jiles- Atherton  model  for  ferromagnetic  hysteresis  [9],  where  hysteresis  is  considered  to 
arise  from  pinning  of  domain  walls  on  defect  sites.  The  most  popular  phenomenological  hysteresis 
model  used  in  control  of  smart  actuators  has  been  the  Preisach  modelfl,  6,  7]. 

A  fundamental  idea  in  coping  with  hysteresis  is  inverse  compensation  [5,  11,  13].  Inverse  com¬ 
pensation  suffers  from  a  couple  of  drawbacks,  like  no  closed  form  and  implementation  difficulties. 
In  this  report,  we  will  investigate  the  control  of  hysteretic  actuator  from  a  different  perspective.  We 
will  study  a  special  class  of  hysteretic  systems  which  have  low  dimensional  mathematical  models. 

To  be  specific,  we  will  focus  on  control  of  a  commercially  available  magnetostrictive  actuator. 
At  low  frequencies,  the  magnetostriction  can  be  related  to  the  bulk  magnetization  through  a  square 
law,  thus  control  of  magnetostriction  is  equivalently  control  of  bulk  magnetization.  We  will  employ 
the  low  dimensional  bulk  magnetization  model[14]  for  the  magnetization  hysteresis.  The  model  is  a 
hybrid  dynamical  system,  whose  switching  depends  on  both  the  state  and  the  control.  Conclusions 
and  future  work  are  provided  in  Section  6. 

This  report  is  organized  as  follows.  In  Section  2  we  present  the  hysteresis  model  and  explore 
its  properties.  In  Section  3  we  formulate  an  optimal  control  problem,  and  show  the  value  function 
satisfies  a  Hamilton- Jacobi-Bellman  equation  (HJB)  of  some  hybrid  form  in  the  viscosity  sense. 
In  Sectin  4,  We  prove  that  (HJB)  admits  a  unique  solution  in  the  class  of  continuous  functions 
to  which  the  value  function  belongs.  We  describe  some  discrete  approximation  schemes  to  (HJB) 
in  Section  5.  This  establishes  the  existence  of  a  solution  to  (HJB)  as  well  as  provides  a  way  for 
suboptimal  control  synthesis.  Finally  future  work  along  the  line  of  this  report  is  provided  in  Section 
6. 


2  Mathematical  Model  of  Hysteresis 

2.1  The  bulk  ferromagnetic  hysteresis  model 

Jiles  and  Atherton  proposed  a  low  dimensional  model  for  ferromagnetic  hysteresis,  based  upon  the 
quantification  of  energy  losses  due  to  domain  wall  intersections  with  inclusions  or  pinning  sites 
within  the  material[9].A  modification  of  the  Jiles- Atherton  model  was  made  by  Venkataraman 
and  Krishnaprasad  with  rigorous  use  of  energy  balancing  principle[15,  14],  and  they  called  it  the 
bulk  ferromagnetic  hysteresis  model.  Also  based  on  the  energy  balancing  principle,  they  derived 
a  bulk  magnetostrictive  hysteresis  model[16,  14].  At  low  frequencies,  the  magnetostriction  can  be 
related  to  the  bulk  magnetization  through  a  square  law[14],  thus  control  of  the  bulk  magnetization 
amounts  to  control  of  the  magnetostriction.  In  this  report,  we  will  study  optimal  control  of  the 
bulk  magnetization  exclusively  to  highlight  the  methodology  of  hysteresis  control.  Extension  to 
control  of  magnetostriction  at  high  frequencies  can  be  done  following  the  ideas  in  [12]. 
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We  will  use  the  bulk  ferromagnetic  hysteresis  model  [15,  14]  in  this  report,  which  has  a  slightly 
different  form  from  the  Jiles-Atherton  model.  We  now  briefly  outline  the  model.  The  bulk  magne¬ 
tization  M  is  comprised  of  a  reversible  component  Mrev  and  an  irreversible  component  Mirr ,  and 
Mrev  is  related  to  Mirr  and  the  anhysteretic  magnetization  Man  by: 

Mrev  =  c(Man  -  Mirr),  (1) 

where  c  is  called  the  reversibility  coefficient,  and  Man  is  given  below. 

For  an  input  field  H  and  a  bulk  magnetization  M,  we  define  He  =  H  +  aM  to  be  the  effective 
field,  where  a  is  a  mean  field  parameter  representing  inter-domain  coupling.  Through  thermody¬ 
namic  considerations,  the  anhysteretic  magnetization  Man  can  be  expressed  as 

Man(He)  =  Ms(coth(—)  -  ■£-) 

a  He 

=  MsC{z),  (2) 

where  £(•)  is  the  Langevin  function,  C(z)  =  coth(z)  —  with  z  =  Ms  is  the  saturation 
magnetization  of  the  material  and  a  is  a  parameter  characterizing  the  shape  of  Man  curve.  Energy 
balancing  yields  the  expressions  for  Mirr  '■ 

dMirr  _  ~Ho(Man  -  M )  dHe 

dH  ~  M(l-c)  dH  ’  U 

where  fiQ  is  the  permeability  of  vacuum,  A;  is  a  measure  for  the  average  energy  required  to  break  a 
pinning  site,  8  =  sign(H),  and 


{0,  dH  <  0  and  M  <  Man 
0,  dH  >  0  and  M  >  Man  . 

1,  else 

The  function  8  is  defined  to  guarantee  that  pinning  always  opposes  changes  in  magnetization,  and 
8  is  defined  to  guarantee  that  the  incremental  susceptibility  is  non-negative.  Since  by  (1), 

M  =  Mirr  +  Mrev  =  (1  —  c)iWjrr  +  cMan,  (4) 


taking  derivative  with  respect  to  H  at  both  sides  of  (  4),  we  have 


dM  dMirr  dMan 

oh  =  {l-c)^nr+c^iir 

From  (2)  and  (3),  we  get  after  some  manipulations, 

dM  to"'  dC£]  +  j<M°"  -  M> 

dH  M(1  _  ssMiSjM.)  _  -  M)  ’ 


where 


dC(z) 

dz 


—  csch 2 (z) . 


(5) 

(6) 
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Equation  (5)  describes  a  switched  nonlinear  system.  In  particular,  letting 


fi(H,  M)  =  c- 


M. 


d/l(z) 


s  dz 


a  —  acM , 


dC(z)  ’ 


)  = 

and  u  =  H,  we  can  rewrite  (5)  as 


ckMs^f  -  y0a(Man  -  M) 
k(a  -  acMs^ft)  +  fi0aa(Man  -  M) 
ckMs^f  +  ^a(Man  -  M) 
k(a  -  acMs^ft)  -  / i0aa(Man  -  M) 


H 

M 


1 


u. 


(7) 


with  each  f,  smooth  in  H  and  M,  and  the  switching  rule  is: 

(  l,u  <  0,  M  <  Man(He )  or  u  >  0,  M  >  Man(He ) 
i  =  <  2,  it  <  0,  M  >  Man(He) 

[  3,u  >  0,  M  <  Man(He) 

Note  the  switching  depends  on  both  (sign  of)  u  and  the  state  variables  H,  M.  We  may  represent 
model  (7)  in  a  more  compact  way.  Let  T={(H,  M)  :  M  =  Man(He)} , 

={(H,  M )  :  M  <  Man(He)},  fi2={(F,  M)  :  M  >  Man(He)}, 

and  denote  fi*  =  fij  U  T,  i  =  1, 2.  Letting  x  =  (H,  M),  we  can  define 


/+(*)  =  S 


1 

fi(x) 

1 

fd(x) 


if  x  e  fi2 


if  x  £  fi  i 


,  and  /_  (x)  = 


1 

fi(x) 

1 

f2(x) 


if  x  e  fii 
if  x  G  fi2 


Since  fi,  1  <  i  <  3,  coincide  on  T,  /+  and  /_  are  well  defined  and  continuous.  We  then  introduce 
the  discrete  control  set  D  =  {1,2}  and  the  continuous  control  sets 

U+  =  {u  :  u  <  0},  U-  =  {u  :  u  >  0}. 

A  control  action  includes  both  the  discrete  mode  control  d  £  D  and  the  continuous  control  u.  Now 
the  model  (7)  can  be  described  as:  at  any  x  G  M2,  if  one  chooses  d  =  1,  then  u  must  be  picked  from 
U+,  and  the  dynamics  is  governed  by: 


X  =  f+(x)u. 

Similarly,  if  d  =  2  is  chosen,  then  u  must  be  picked  from  £/_,  and  the  dynamics  is: 


(8) 


X  =  f-(x)u.  (9) 

The  state-dependent  switching  has  now  been  incorporated  into  the  definitions  of  /+,  /_.  Note  the 
model(8),(9)  is  a  Duhem  hysteresis  model  [17].  The  Duhern  model  characterizes  a  class  of  rate- 
independent  hysteresis  models  with  input  v(-)  and  output  w(-),  with  dynamics  depending  on  the 
sign  of  v.  To  be  precise, 

Co  =  gi(y,u)(y)+ -  g2(y,u)(y)- ,  (10) 
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where  (v)+  =  max{0,  h}  and  ( v ) 


with  the  switching  rule: 


=  max{0,  — u}.  Denoting  u  =  v,  we  can  write  (10)  as 


v 

LO 


1 

9i(v,u) 


u, 


(11) 


(  1,  u  <  0, 
\  2 ,u>  0, 


Remark:  Smith  and  Horn  [10]  proposed  a  model  for  ferroelectric  hysteresis  analogous  to  the 
Jiles- Atherton  model  for  ferromagnetic  materials.  The  model  of  Smith  and  Horn  carries  the  same 
structure  as  that  of  (7),  thus  the  approach  presented  in  this  report  is  fully  applicable  to  control  of 
actuators  made  of  ferroelectric  materials.  This,  in  some  sense,  justifies  the  title  of  the  report. 

In  the  next  two  subsections,  we  will  derive  some  properties  of  the  model  (7),  which  will  be  used 
in  the  analysis  later. 


2.2  Boundness  of  ft 


Lemma  1:  C(z)  satisfies: 


n  dC(z)  1 
<  dz  ~  3’ 
|£(z)|<l. 


The  proof  of  Lemma  1  can  be  found  in  Appendix  A. 

Proposition  2 :If  the  parameters  satisfy: 

m  acMs 

T\  ■=  a - >  0, 


3 

m  ,  ,  acMs . 

T2  :=  k(a - - — )  -  2 p,0aaMs  >  0, 


then  0  <  fi  <  Cf,i  =  1,2,3  for  some  constant  Cf  >  0. 
Proof.  By  (12)  and  (14) 


m  acMs  dC(z) 

0  <  Tj  =  a - - —  <  a  —  acMs — - —  <  a. 


dz 


We  rewrite  f\  as 


h  =  —  + 


a  a(a  —  acMs^z£p-) 
and  note  that  it  is  a  nondecreasing  function  of  ■  Since 

fi  =  0  when  =  0, 


fi  = 


cMs  ^  ___! _  dC{z)  _  i 


3a  —  acA'L 


=  :  C i  when 


dz  3’ 


(12) 

(13) 


(14) 

(15) 
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we  get 


0  <  h  <  Ci. 


The  function  f2  can  be  written  as 

1  ^  ka  1 

a  k(a  -  acMs  )  +  Mo aa{Man  -  M) 

' - — - V - ' 

=:T 

From  the  model  (7),  when  f2  is  selected,  Man  —  M  <0.  Since  magnitudes  of  both  Man  and  M 
must  be  less  than  Ms,  Man  —  M  >  —2 Ms.  These  facts  together  with  (12)  yield 

0  <  T2  <  T  <  ka. 


Therefore 


0  </2< 


ka  —  T2 
aT2 


=-c2. 


Similarly  we  can  show  0  <  fs  <  C2-  Picking  Cf  =  max{(7i,  C2},  we  have  0  <  fi  <  Cf  for 
z  =  1,2,3.  □ 

Remark:  Conditions  (14)  and  (15)  are  satisfied  for  typical  parameters.  For  example,  if  we  take 
the  identified  parameters  in  [14],  a  =  1.9  x  10-4,  a  =  190,  k  =  48  Tesla ,  c  =  0.3,  Ms  =  7.9  x  105 
Amp/meter  and  po  =  47r  x  10  '  Henry/meter ,  we  calculate  T\  =  174.99,  T2  =  8.40  x  103. 


2.3  Lipshitz  continuity  of  the  model 

Proposition  3:  /+  and  /_  are  Lipshitz  continuous  with  Lipshitz  constant  L.  Define  f+(x,u)  = 
f+{x)u ,  f-(x,u)  =  f-(x)u.  If  U. |_  =  {u  :  0  <  u  <  uc}  and  U-  =  {it  :  —  uc  <  u  <  0}  for  some 
uc  >  0,  then  \/u  G  U+(u  G  U-,  resp.),  f+(x,u)  (f-(x,u),  resp.)  is  Lipshitz  continuous  with  respect 
to  x  with  Lipshitz  constant  Lq  =  Luc. 

Remarks: 


•  The  physical  interpretation  for  |it|  <  uc  is  the  operating  bandwidth  constraint  on  the  rnagne- 
tostrictive  actuator. 

•  In  the  rest  of  the  report,  we  will  use  f+{x)u  ( f-{x)u ,  resp.)  and  f+(x,u)  (f-(x,u),  resp.) 
interchangeably. 


Proof.  We  first  prove  /_  is  Lipshitz  continuous  with  Lipshitz  constant  L.  We  discuss  three 
cases: 


•  Case  I:  Both  xi,x2  G  fh.  In  this  case,  mode  1  is  active,  and  thus 


df-(x) 

dx 


0  0 

dfi(H,M)  dfi(H,M) 
dH  dM 


(16) 
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It  can  be  shown,  after  some  manipulations,  that  |  1  <  Cl,  1  1  —  «Ci  for  some 

C±  >  0.  Therefore  |  1  —  -^l  f°r  some  Li  >  0,  and  the  following  holds: 

|/_(xi)  -  /_(x2)|  <  Li\xi  -  x2\. 

•  Case  II:  Both  xi,x2  G  fi2-  In  this  case,  mode  2  is  active.  Following  similar  steps  as  in  Case 

I,  we  can  show  |  |  <  C2  for  some  L2  >  0  and  therefore 

|/_(xi)  -  /_( x2)\  <  L2\xi  -  x2\. 

•  Case  III:  x\  G  Oi ,  a?2  £  ^2-  Then  there  exist  xo  G  T,  such  that  the  line  segment  connecting 
x'i  and  x2  intersects  T  at  xo-  We  express  xq  =  0xi  +  (1  —  0)x 2  with  0  <  9  <  1.  Thus 

-  f-(x2)\  =  |/-(xi)  - /_(x0) +  /_(x0)  - /_(x2)| 

<  Ti|xi  -  x0|  +  T2|x0  -  X2I 

=  L\(l  -  6)\x\  -  x2\  +  L26\xi  -  x2\ 

<  L_|xi  -  x2|, 

with  L_  =  max{Li,L2}. 

Following  exactly  the  same  arguments,  we  can  show  , 

|/+(xi)  -  /+(x2)|  <  T+|xi  -  x2|,  Vxi,x2. 

We  conclude  the  first  part  by  taking  L  =  max{L_,  L+}.  The  rest  of  the  proposition  follows  trivially. 

□ 

3  Optimal  Control:  (HJB)  and  Viscosity  Solutions 

Dynamic  Programming  Principle  (DPP)  is  one  of  the  most  important  approaches  in  optimal  control. 
When  the  value  function  is  smooth,  we  can  derive  the  Hamilton- Jacob i-Bellman  equation  (HJB), 
and  in  many  cases,  solving  HJB  amounts  to  solving  the  optimal  control  problem.  The  value 
function  however,  in  general,  is  not  smooth  even  for  smooth  systems,  not  to  mention  for  a  hybrid 
system,  like  that  in  our  model.  Crandall  and  Lions [4]  introduced  the  notion  of  viscosity  solutions 
to  Hamilton- Jacobi  equations.  This  turned  out  to  be  a  very  useful  concept  for  optimal  control  since 
value  functions  of  many  optimal  control  problems  do  satisfy  the  HJB  in  the  viscosity  sense.  And 
under  mild  assumptions,  uniqueness  and  existence  results  for  viscosity  solutions  hold. 

We  will  explore  this  approach  for  control  of  smart  actuators.  This  report  is  aimed  at  providing 
some  flavors  of  this  approach  through  the  example  of  infinite-time  horizon  optimal  control  problem. 
We  will  study  the  properties  of  the  value  function,  derive  the  Dynamic  Programming  Principle  and 
show  the  value  function  indeed  satisfies  (HJB)  of  a  special  form. 
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3.1  Optimal  control  problem 


For  ease  of  presentation,  we  rewrite  the  model  (8), (9)  as 


x  =  f(x,u,d) 


f+(x,  u)  if  d  =  1,  u  £  U+ 
/_  (x,  u)  if  d  =  2,  u  G  U- 


(17) 


We  require  «(•)  to  be  measurable.  This  together  with  Proposition  3  guarantees  that,  for  any 
initial  condition  x  and  any  admissible  control  pair  a(-)  :=  {d(-),  «(•)},  (17)  has  a  unique  solution 
x(-)(the  dependence  on  x  and  «(•)  is  suppressed  when  no  confusion  arises). 

Define  the  cost  functional  with  initial  condition  x  and  control  a(-)  as 

pOO 

J(x,  «(•))=  /  l(x(t),u(t))e~xtdt,  (18) 

Jo 

with  A  >  0.  The  optimal  control  problem  is:  given  initial  condition  x,  find 

V(x)  =  inf  J(x,  a(-)),  (19) 

“(•) 

and  if  V(x)  is  achievable,  find  the  optimal  control  a*(-). 

We  make  the  following  assumptions  about 


•  (Tli):  l(x,u)  continuous  in  x  and  u,  l(x,u )  >  0,  \/x,u ; 

•  (A2):  Z(0,0)  =0; 

•  (A3):  | l(xi,u)  —  l(x 2, it) |  <  Ci(  1  +  |xi|  +  |a72 1) |£Ci  —  X2\,  Vu  for  some  Ci  >  0. 


Note  (A3)  includes  the  case  of  quadratic  cost. 


3.2  Properties  of  the  value  function 

Proposition  4  [Local  Boundness]:  Under  assumptions  (Ai)  —  (A3),  VA  >  0,  V(-)  is  locally 
bounded,  i.e.,  Vi?  >  0 ,3Cr  >  0,  such  that  |F(.r)|  <  Cr  \/x  G  B(0,R)  :=  {x  :  |x|  <  i?}. 

Proof.  First  note  since  ?(•,•)  is  nonnegative,  V(x)  >  0  Vx.  Take  u(t)  =  0,then  x(t)  =  x.  Let 
aft)  =  {d(t),u(t)}  where  d{t)  =  1.  We  have 

V(x)  <  J(x,  «(•))  =  /  l(x,0)e~xtdt 

Jo 

l(x,  0) 

A  ' 

By  (A2)  and  (A3),  l(x,  0)  <  (7/(1  +  R)R,  and  the  proof  is  complete  with  Cr  :=  c'd1+^)^  _  □ 
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Proposition  5  [Locally  Lipshitz  Continuity]:  Under  assumptions  (^4i)  —  (.A3),  VA  >  2Lq 
with  Lq  as  defined  in  Proposition  3,  V (■)  is  locally  Lipshitz,  i.e.,  VR  >  0,3Lr  >  0,  such  that 
|P(.xi)  -  V(x2)\  <  Lr\xi  -  x2\  Mxi,x2  G  B(0,R). 

Before  we  prove  Proposition  5,  we  first  prove  a  lemma  regarding  the  solution  to  (17). 

Lemma  6:  Let  x\{-) ,  x2{-)  he  solutions  to  (17)  under  some  admissible  control  «(•)  =  {d(-),  ?/(•)} 
with  initial  condition  x\,x2  respectively.  Then 


1. 


xi(t)  -  x2(t) |  <  eLot\xi  -  x2\; ; 


(20) 


2. 


C 


|*i (*) I  <  \xi\eLot  +  j-(eLot  -  1), 
L  0 


(21) 


where  C  =  max^  |/(0,  uc ,  d)\. 

Proof.  1.  Denote  the  sequence  of  mode  switching  times  as  {ti,i  =  0, 1,  •  •  •  }  with  to  =  0,  and 
the  mode  during  [ti,  U+i)  as  dj.  Then  Vf  G  [0,ti), 

^|*i (t)  -  x2(t)\2  =  2(xi (t)  -  x2(t))  ■  (f(xi(t),u(t),do)  ~  f(x2(t),u(t),do) 

<  2L0\xi(t)  -  x2(t)\2, 

where  the  inequality  comes  from  Proposition  3.  Integrating  both  sides  from  0  to  t, 

\xi(t)  -  x2(t)\2  <  \xi  -  x2\2  +  2L0  f  |xi(s)  -  x2(s)\2ds, 

Jo 

and  by  Gronwall  inequality, 

\xi (t)  -  x2(f)|2  <  |xi  -  x2\2e2Lot, 

from  which  (20)  follows.  Now  Vi  G  [t±,t2),  taking  x\[t\),x2[t{)  as  initial  conditions,  we  follow  the 
above  procedures  and  get 

\xi(t)  -  x2(t)\  <  |xi(fi)  -  x2(ti)|eLo(^*l) 

<  Ixi  -x2|eLo(t_tl)eLo(tl_0) 


=  xi-x2e 


Lot 


Using  the  same  argument  successively,  we  can  show  that  (20)  holds  Vi  >  0. 


2.  Vx'i  (t)  0,  we  can  write 


< 


=  *i(*)  ■  f(xi(t),u(t),d(t)) 

Xi (i)  ■  (f(0,u(t),d(t))  +  f(xi (i),u(i),d(i))  -  f(0,u(t),d(t))) 
C|*i(i)|  +  L0\x!(t)\2, 


from  which  we  obtain 


—  |xi(t)|  <  C  +  L0\xi(t)\. 
at 

Integrating  it  from  0  to  t  and  then  apply  Gronwall  inequality,  we  have  (21).  □ 

Proof  of  Proposition  5.  For  e  >  0,  let  cF(-)  =  {cZe (•) ,  rte(-)}  be  e— optimal  for  X2,  he., 


V(x2)  >  J(x2,ae(-))  -  e. 


Since  V(x\)  <  J(x i,ae(-)),  we  have 


V(xi)-V(x2)  <  J(xi,a£(-))  -  J{x2,ae{-)) 


rOC 

<  /  e~xt\l(xi(t),ue(t))  -  l(x2{t),ue(t))\dt  +  e 

Jo 

roo 

<  /  e~xtCi(l  +  \x1(t)\  +  \x2(t)\)\xi(t)  -  x2{t)\dt  +  e, 
Jo 


where  the  last  inequality  is  from  (.A3).  Using  Lemma  6, 

r 00  o/o 

V(xi)-V(x2)  <  Ci\xi-x2\  ((—  +  |xi|  +  |x2|)e(2I,0“A)t 

Jo  Lo 

+(1  -  ^f-)e<'L°~x)t)dt  +  e 
Lo 

(n  ,  (M  +  |x2|)  „  ,  ,  , 

—  {Co  H - 7 - vy - )Ci\x\  —  x2\  +  e 

A  —  ZL/  0 
<  LR\x1-x2\  +  e, 

where  Co  is  a  constant  and  LR  :=  (Co  +  a-2L0  Since  e  is  arbitrary,  we  have 

U(xi)  -  V{x2)  <  Lr \xi  -  x2|. 

But  x\  and  x2  are  symmetric,  we  must  also  have 


Therefore 


V{xi)  -  V(x2)  <  Lr \xi  -  x2\. 


\V{x\)  -  V(x2)\  <  Lr \xi  -  x2\. 


Remark:  Proposition  2  can  be  exploited  to  yield  sharper  estimates  for  |xi(t)  —  x2(t)\  and 
|xi(t)|,  as  shown  in  the  next  lemma.  This  might  be  used  to  weaken  the  assumptions  in  Proposition 


Lemma  7:  Let  aq(-),  x2(-)  be  solutions  to  (17)  under  some  admissible  control  a(-)  =  {d(-),  ?/(•)} 
with  initial  condition  x\,x2  respectively.  Then 


|a:i(t)  -  x2(t) |  <  |xi  -  x2\  +  2 L0t; 
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2. 


(23) 


xi (t)  -  xi\  <  C0t , 


where 


C0  =  uc 


1 

Cf 


Proof. 1.  For  x\ (t)  /  x2  (t), 

\x\ (t)  -  x2(t)\-^\x1(t)  -  x2(t) I 

= 

=  (xx(t)  -  x2(t))  ■  (f(xi(t),u(t),d(t))  -  f(x2(t),u(t),d(t))) 
<  2L0\xi(t)  -  x2(t)\, 

where  the  last  inequality  is  from  Proposition  3.  Therefore  we  have 

-^|aq(t)  -  x2(t)\  <  2 L0, 

from  which  (22)  follows. 

2.  (23)  is  a  straightforward  consequence  of  Proposition  2.  □ 


3.3  Dynamic  programming  principle 

Proposition  8  [Dynamic  Programming  Principle]:  Assume  (Ai)  —  (.A3),  A  <  2Lq.  Denote 
any  admissible  control  pair  (d(-),tt(-)}  as  «(•).  The  following  is  true: 

V(x)  =  ini{[  e~Xsl(x(s),  u(s))ds  +  e~xtV(x(t))},  Vt  >  0.  (24) 

«(•)  Jo 

Proof.  Denote  the  right  hand  side  of  (24)  as  W{x).  Note  under  the  assumptions,  Vt  >  0,  V(x) 
and  W(x)  are  locally  bounded,  i.e. ,  for  any  bounded  x,  V(x)  <  00,  W(x)  <  00.  We  will  first  show 
V (x)  >  W (x)  and  then  the  converse. 

1.  For  e  >  0,  let  cP(-)  be  e— optimal,  i.e., 

J(x,ae(-))  <  V(x)  +  e. 


Now 


rt  r  00 

J(x,ae(-))  =  /  e~Xsl(x(s),  u€(s))ds  +  /  e~Xsl(x(s),ue(s))ds 

Jo  Jt 

rt  roc 

=  /  e~Xsl(x(s),  ue(s))ds  +  e~xt  /  e~Xsl(x(s  +  t),  u£(s  +  t))d-s 

Jo  Jo 
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Let  x(s)  =  x(s  +  t),a(s)  =  a£(s  +  t),  we  have 

/*oo  /*oo 

/  e~Xsl(x(s  +  t),  u£(s  +  t))ds  =  /  e~Xsl(x(s),u(s))ds 

Jo  Jo 


Therefore 


=  J(x(t),u(-))  >  V(x(t)). 

V[x)  +  e  >  W(x), 


and  since  e  is  arbitrary,  V(x)  >  W(x). 

2.  For  e  >  0,  pick  ao(-)  =  {do(-),  wo(-)}  such  that 

[  e~Xsl(x(s),  uo(s))ds  +  e~xtV(x(t))  <W(x)  +  e. 

Jo 

Then  pick  ai(-)  =  {di(-), «i(-)}  such  that 

J(x(t),ai(-))  <  V(x(t))  +  e. 

Now  define  a(-)  =  {d(-),u(-)}  by 

a(s)  = 

We  then  have 

rt  /*oo 

W(x)  +  e  >  /  e~Xsl(x(s),  uo(s))ds  +  e~xt{  /  e~x^l(x{s)1ui{s))ds  —  e) 

Jo  Jt 

n  oo 

>  /  e~Xsl(x(s),u{s))ds  —  e 

Jo 

=  J(x,a(-))  —  e  >  V(x)  —  e, 

which  implies 

W(x)  >  V(x )  -  2e. 

Since  e  is  arbitrary,  we  have  W(x)  >  V(x).  □ 


ao(s),  s  <  t 
ai(s  —  t),  s  >  t. 


3.4  Hamilton-Jacobi-Bellman  equation 

In  this  subsection,  we  will  show  that  the  value  function  V(-)  satisfies  the  HJB  equation  in  the 
viscosity  sense.  Viscosity  solutions  to  Hamilton- Jacobi  equations  were  first  introduced  by  Crandall 
and  Lions  [4],  Here  we  use  one  of  the  three  equivalent  definitions  of  viscosity  solutions  [3]: 

Definition[Viscosity  Solution]:  Let  W  be  a  continuous  function  from  an  open  set  O  G  Mn 
into  M  and  let  DW  denote  the  gradient  of  W  (when  W  is  differentiable).  We  call  W  a  viscosity 
solution  to  a  nonlinear  first  order  partial  differential  equation 

F(x,W(x),DW(x))  =  0,  (25) 

provided  V(/>  G  C'1(0), 
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( Viscosity  Subsolution  )  if  W  —  (f>  attains  a  local  maximum  at  xq  G  O ,  then 


F(x0,W(x0),D(j>(xo))  <0. 

•  ( Viscosity  Supersolution  )  if  W  —  (f>  attains  a  local  minimum  at  xo  G  O ,  then 

F(x0,W(x0),D(j>(xo))  >0. 


Viscosity  solutions  have  a  couple  of  nice  properties[3,  4].  We  mention  one  elementary  property 
here( consistency  with  the  notion  of  classical  solution ),  that  is:  1)  any  classical  solution  to  (25)  is  a 
viscosity  solution;  2)  the  viscosity  solution  satisfies  (25)  in  the  classical  sense  at  any  point  where  it 
is  differentiable. 

We  now  present  the  first  main  result  of  this  report:  the  value  function  V(-)  satisfies  a  Hamilton- 
Jacobi-Bellman  equation  of  a  special  form  in  the  viscosity  sense. 

Theorem  9  [Hamilton-Jacobi-Bellman  Equation]  Assuming  (.Ai)  —  (.A3),  A  >  2Lq,  the 
value  function  V (x)  is  a  viscosity  solution  to  the  following  equation: 

( HJB )  A W{x)  +  max{  sup  {— uf+{x)  ■  DW(x)  —  l(x,u)},  sup  {— uf-(x )  •  DW(x)  —  l(x,u)}}  =  0. 

u&U+  u&U- 

(26) 


Remark:  we  may  replace  sup  in  (26)  by  max  since  U-  and  £/+  are  compact. 

Proof.  1.  We  first  show  V(-)  is  a  viscosity  subsolution.  For  any  u  G  £/_,  take  «(•)  =  {d(-),tt(-)} 
with  d(t)  =  2,  u(t )  =  u  Vt.  From  (24),  for  any  t  >  0 

V(x)  <  f  l(x(s),  u)e~Xsds  +  e~xtV(x(t))i 
Jo 

which  we  rewrite  as 

V (x(t))  —  V(x)  +  [  l(x(s),  u)e~Xsds  +  V{x[t)){e~xt  —  1)  >  0.  (27) 

Jo 

Now  suppose  that  V  —  <j)  with  f>  G  C'1(M2)  has  a  local  maximum  at  x,  then  since  V  is  continuous, 

V(x(t))  —  (f(x(t))  <  V(x)  —  <j)(x), 
for  t  sufficiently  small.  This  together  with  (27)  implies 

4>(x(t))  —  4>{x)  +  [  l{x{s),  u)e~Xsds  +  V(x(t))(e~xt  —  1)  >  0.  (28) 

Jo 

Divide  (28)  by  t  and  let  f  — >  0,  we  obtan 

uf-(x)  ■  D<j>(x)  +  l(x,  u )  —  XV (x)  >0,  Vu  G  U-, 


i.e., 


XV (x)  +  sup  {—uf-(x)  ■  D<f)(x)  —  l(x,u)}  <  0. 

ueU- 
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Similarly,  we  have 


\V(x)  +  sup  {— uf+(x)  ■  D<t>{x)  —  l(x,u)}  <  0. 
ueU+ 

Therefore 

XV (x)  +  max{  sup  {—uf-(x)  ■  D(j>(x)  —  l(x,u)}, 

u&U- 

sup  {— uf+(x)  ■  D<f>(x )  —  l(x,u)}}  <  0.  (29) 

ueu+ 


2.  The  proof  of  supersolution  is  much  more  technically  involved.  We  mainly  follow  the  approach 
in  Bardi  and  Capuzzo-Dolcetta[2]. 

For  t  >  0,  e  >  0,  we  can  find  a(-)  =  {d(-),  ?/(•)},  such  that 

V(x)>  f  l(x(s),  u(s))e~Xsds  +  e~xtV(x(t))  —  te.  (30) 

Jo 

Since  by  (A3), 

| l(x(s),u(s))  -  l(x,u(s)) |  <  Ci(  1  +  |x(s)|  +  |x|)|x(s)  -  x\, 

and  by  Lemma  7, 


|a;(s)  —  x\  <  Cos, 


we  then  have 


f  l(x(s),u(s))e  Xsds=  f  l(x,u(s))e  Xsds  +  o(t). 
Jo  Jo 


And  by  (30), 

V(x)>  f  l{x,  u(s))e~Xsds  +  e~xtV(x(t))  —  te  +  o(t).  (31) 

Jo 

Now  suppose  that  V  —  4>  with  <f>  E  C'1(M2)  has  a  local  minimum  at  x,  then  since  V  is  continuous, 

V(x(t))  —  (f>{x{t))  >  V(x)  —  4>(x), 
for  t  sufficiently  small.  This  and  (31)  imply 


(j>{x)  —  cf>(x(t))  >  f  l(x,u(s))e  Xsds  —  (  1  — e  xt)V(x(t))  —  te  +  o(t). 

Jo 


(32) 


We  also  have 


<t>(x)  ~  (j>(x{t))  =  -  D<j>(x(s))  ■  f{x(s),u{s),d{s))ds 

Jo 

=  —  D(f>(x)  ■  f(x,  u(s),  d(s))ds  +  o(t), 

Jo 


(33) 
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since 


\f(x(s),u(s),d(s))  -  f(x,u(s),d(s))\  <  L0\x(s)  -  x\ 

<  CqL0s, 


and 


I D(/>(x(s))  -  Dcj)(x) |  <  ujD<f,(\x(s)  -  z|), 
where  a JD<p{')  is  the  modulus  of  continuity  of  D(f>. 

Combining  (32)  and  (33)  gives  rise  to 

f  \—D(j)(x)  ■  f  (x ,  u(s) ,  d(s))  —  l(x,u(s)))ds  +  f  (1  —  e~Xs)l(x,  u(s))ds 
Jo  Jo 

+(1  -  e~xt)V (x(t))  >  -te  +  o(t). 


(34) 


Since  U-  U  U+  is  compact,  l(x,u(s))  <  Mx,  Vs  >  0,  for  some  constant  Mx  depending  on  x.  Thus 

f  (1  —  e~Xs)l(x,u(.s))ds  =  o(t). 

Jo 

Since 

—Dcj>(x)  ■  f(x,u(s),d(s))  —  l(x,u(s))  <  max{  sup  {—uf-(x)  ■  D<f>(x )  —  l(x,u)}, 

U£U- 

sup  {—uf+(x)  •  D(j>(x )  —  l(x,u)}}  =:  Q(x,  D(j>(x)), 

uGU+ 

we  obtain  from  (34) 

f  Q(x,  D4>[x))ds  +  (1  —  e~xt)V(x(t))  >  —  te  +  o(t).  (35) 

Jo 

By  dividing  (35)  by  t  and  letting  t  — >  0, 

XV (x)  +  Q(x ,  Dcj)(x))  >  —e. 

Since  e  is  arbitrary,  we  get 

XV (x)  +  max{  sup  {— uf-(x)  •  D(j>(x)  —  l(x,u)}, 

u&U- 

sup  {— uf+(x)  ■  D(j)(x)  —  l(x,u)}}  >  0.  (36) 

u£U+ 


Combining  (29)  and  (36)  yields  the  desired  result. □ 


14 


4  Uniqueness  of  (HJB) 


We  would  like  to  characterize  the  value  function  V(-)  as  a  unique  solution  to  (HJB).  The  uniqueness 
result  basically  comes  from  Theorem  1.5  in  [8].  In  [8],  the  author  gave  only  a  sketch  of  proof.  Here 
for  completeness,  we  will  provide  the  full  proof. 


Before  stating  the  theorem,  we  need  first  identify  structural  properties  of  our  (HJB).  We  rewrite 
(26)  as: 

A W(x)  +  H{x,  DW(x ))  =  0,  (37) 


where 


H(x,p)  =  max{  sup  {—  p  •  /_(#,  u)  —  fix,  it)},  sup  {—  p  ■  f+(x,  u)  —  fix,  it)}}. 
uGU-  u£U+ 


Proposition  10:  Assume  (A3),  H(x,p)  satisfies  the  following: 


\H(xi,p)  -  H(x2,p)\  <  CR(1 +  \p\)\x1  -  x2\,  Vxi,x2  e  B(0,R),  \/p  (38) 

\H(x,pi)  -  H(x,p2)\  <  CQ\pi -p2\,\/x,  Vpi,p2  (39) 

for  some  Cr  >  0,  Co  >  0,  with  Cr  dependent  on  R. 

Proof.  1.  Without  loss  of  generality,  suppose  u\  £  U-  attains  the  maximum  in  H(xi,p).  Since 
H(x2,p )  >  -p  ■  f-(x2,ui)  -  l(x2,ui), 

H(xi,p)  -  H(x2,p)  <  -p- f-(xi,ui) -l(xi,ui)+p- f-(x2,ui)  +  l(x2,ui) 

<  \p\Lo\xi  -  x2 1  +  Ci(  1  +  |xi|  +  |x2|)|a;i  -  x2\ 

<  CR(  1  +  \p\)\xi  -  x2\, 

where  Cr  is  a  constant  dependent  on  R.  By  symmetry,  we  have  H(x2,p)  —  H(xi,p)  <  Cr(  1  + 
\p\)\xi  -  x2\. 


2.  Without  loss  of  generality,  suppose  u\  £  U-  attains  the  maximum  in  H(x,pi).  Since 
H{x,p2 )  >  -p2  •  f-(x,ui)  -  l(x,ui), 

H(x,pi)  -  H(x,p2)  <  -pi-f-(x,ui)-l(x,ui)+p2-f-(x,ui)  +  l(x,ui) 

=  ~f-(x,Ul)  ■  {jp!  -p2) 

<  Cq  \pi  —  p2\, 

where  the  last  inequality  is  from  boundness  of  /_.  Again  by  symmetry,  we  have  the  other  half  of 
the  inequality.  □ 

Remark:  As  we  have  seen  above,  despite  the  hybrid  structure  of  our  physical  model,  H{x,p) 
enjoys  nice  structural  properties,  which  enables  us  to  prove  the  uniqueness  result. 
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From  Proposition  5,  we  know  that  the  value  function  V(-)  of  our  optimal  control  problem 
belongs  to  the  following  class 

V(R2)  =  {W(-)  : \W{x1)-W{x2)\  <C{l  +  R)\x1-x2\yx1,x2eB(0,R)}. 

Note  W(-)  G  P(M2)  implies 

W(-)  G  Q(M2)  =  (W(-)  G  C(R2)  :  sup  -  2  <  oo}. 

JR2  1 

The  following  theorem  is  adapted  from  Theorem  1.5  in  [8]. 

Theorem  11:  Assuming  (Ai)  —  (A3),  A  >  2Lq,  if  (37)  has  a  viscosity  solution  in  V(R2),  it  is 
unique. 


Proof.  Without  loss  of  generality,  we  take  A  =  1.  Let  G'P(M2)  be  viscosity  solutions 

to  (37).  For  e  >  0,  a  >  0,  m  >  2,  define 

$(t,  y )  =  W(x)  —  V(y)  —  -\x  —  y\2  —  a(<  x  >m  +  <  y  >m) 


where 

<  x  >:=  (1  +  |t|2)  2 . 

Since  W{-),V{-)  gQ(M2), 


lim  $(t,  y)  =  —00. 

|rr|+|j/|->oo 

And  since  <!>(•,  •)  is  continuous,  there  exists  (xo,yo)  such  that  $  attains  the  global  maximum.  First 
we  need  to  get  an  estimate  for  |to|,  |yo|  and  |to  —  yo\. 

From  T(0,0)  <  $(x0,y0), 

W( 0)  -  F(0)  -  2a  <  W(x 0)  -  V(y0)  -  ^\x0  -  y0\2  -  a(<  x0  >m  +  <  y0  >m). 


W(-),V(-)  GQ(M2)  leads  to 


<  ,t0  >m  +  <y0  >m<  Ca(l+  <  t0  >2  +  <  y0  >2), 

where  Ca  is  a  constant  independent  of  e  (but  dependent  on  a).  Since  m  >  2,  there  exists  Ra  >  0 
(independent  of  e),  such  that 

ko|  <  Ra,  \yo\  <  Ra- 


From  $(t0,t0)  +  ®(y0,yo)  <  24>(t0,  2/0), 

^\xo~yo\2  <  W(x0)  -  W(y0)  +  V(x0)  -  V(y0) 
<  C(1  +  Ra)\x0  -  y0\, 

where  the  last  inequality  comes  from  G'P(M2).  Therefore  we  get 

\%o  -  2/o I  <  tC'a, 


(40) 
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with  C'a  depending  on  a  only. 


Now  define 


<t>(x)  =  V{yo)  +  ~\x  -  yo\2  +  a(<  x  >m  +  <  y0  >m ), 
e 

V’(y)  =  W(x o)  -  ^|x0  -  y\2  -  a(<  x  >m  +  <  y0  >m). 
Since  (W  —  <^>)(-)  achieves  maximum  at  xo, 

MV(x0)  +  H(x0,D(f>(x o))  <  0, 


i.e., 

2  _9 

W(x o)  <  -H(x0,  ~(x0  -  y0)  +  am  <  x0  >m  x0).  (41) 

Similarly,  since  (V  —  (/>)(•)  achieves  minimum  at  yo,  we  obtain 

V(yo)  >  - H(y0 ,  ^(x0  -  y0)  -  am  <  y0  >m~ 2  y0).  (42) 

Subtracting  (42)  from  (41),  we  have 

2  2 

W(x0 )  -  V(yQ)  <  H(y0,  ~(xQ  -  y0)  -  am  <  y0  >m~2  y0)  -  H(x0,  ~(xQ  -  yo)  +  am  <  x0  >m~ 2  x0). 
From  (39) 

2  _9  2  ,  _1 
H(yo,  ~(x0  -y0)  -  am  <  y0  >m  yo)  <  H(y0,  ~(x0  -  y0))  +  aC0m  <  y0  >”'  , 

2  ,  _9  2 
-H(x0,-(x0  -  y0)  +  am  <  x0  >m  x0)  <  -H(x0,  ~(x0  -  y0))  +  aC0m  <  x0  >m  , 


and  therefore 

W(x 0)  -  V(y0)  <  H(y0,^(x 0  -  yo))  -  H(x0,^(x0  -  yo))  +  aC0m(<  x0  >m_1  +  <  yo  >m~1) 

<  CRa(l  +  ^\x0  -  yoPko  -  Vo\  +  aC0m(<  x0  >m_1  +  <  yo  >m_1) 
where  the  last  inequality  follows  from  (38). 

Now  fix  a,  construct  a  sequence  {e*,}  with  linr^^  ej~  =  0.  We  denote  the  corresponding 
maximizers  of  as  ( xok,yok )•  Since  Vfc,  (xok,  yok)  G  B(0,Ra),  by  extracting  a  subsequence  if 
necessary,  we  get 

lim  ( x0k,Vok )  ->•  (xa,ya)  €  B(0,Ra).  (43) 

k — »oo 

Also  from  (40),  we  have  xa  =  ya.  For  each  e*,,  from  <h(.T,x)  <  <h(xo,yo)) 

W(x)  —  V (x)  —  2a  <  x  >m  <  W(xQ)  -  V(y0)  ~  —  \x0  -  y0\2  -  a(<  x0  >m  +  <  y0  >m) 

ek 

<  W(x 0)  -  V(y0)  -  a(<  x0  >m  +  <  y0  >m) 

<  CRa{  1  +  ~ |x0  -  yoDko  -  yo\  +  aC0m(<  x0  >m_1  +  <  y0  >m_1) 
—a(<  x0  >m  +  <  y0  >m) 
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and  letting  k  — »  oo, using  (40) 

W(x)  —  V(x)  <  2a(C\ym  <  xa  >m_1  —  <  xa  >m )  +  2 a  <  x  >m  . 
Since  Cm  <  xa  >m_1  —  <  xa  >m<  C"  for  some  C"  >  0, 

W(x)  -  V(x)  <  2 a(C"+  <  x  >m). 


Letting  a  — >  0,  we  get 


W(x)  -  V(x )  <  0,  V.x, 


Since  W  and  V  are  symmetric,  we  also  have  V(x)  —  W{x)  <  0  V.x.  Thus  we  get  W(x)  =  V{x ),  V.x. 

□ 


5  Discrete  Approximation  Schemes 


The  approximation  will  be  accomplished  in  two  steps.  First  we  approximate  the  continuous  time  op¬ 
timal  control  problem  by  a  discrete  time  problem,  derive  the  discrete  Bellman  equation  (DBE),  and 
show  the  value  function  of  the  disrete  problem  converges  to  that  of  the  continuous  problem  locally 
uniformly.  Following  [2],  we  call  this  step  “semi-discrete”  approximation.  Then  we  indicate  how 
to  further  discretize  (DBE)  in  the  spatial  variable,  which  is  called  “fully-discrete”  approximation. 
The  approaches  we  take  here  follows  closely  those  in  [2] (Chapter  VI  and  Appendix  A). 


5.1  Semi-discrete  approximation 

Consider  a  discrete  time  optimal  control  problem  obtained  by  discretizing  the  original  continuous 
time  one  with  time  step  h  G  (0,  4).  The  dynamics  is  given  by 

x[n]  =  x[n  —  1]  +  hf(x[n  —  1], u[n  —  1  ],d[n  —  1]),  x[0]  =  x,  (44) 

and  the  cost  is  given  by 

OO 

Jh(x,a[-])  =  ^2  hl(x[n],  u[n])(l  —  Xh)n,  (45) 

n=0 

where  a[-]  =  {d[-],w[-]}  is  the  control.  The  value  function  is  defined  to  be 

Vh(x)  =  inf  Jh(x,  «[•]).  (46) 

The  following  lemma  is  analogous  to  Lemma  6. 

Lemma  12:  Let  x i [-] ,  [•]  be  solutions  to  (44)  under  some  admissible  control  «[•]  =  {d[-],w[-]} 

with  initial  condition  X\,X2  respectively.  Then 

1. 

\xi[n]  -  x2[n] |  <  (1  +  hL0)n\xi  -  x2\;  (47) 
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2. 


(48) 


xi[n\\  <  (1  +  hL0)n\xi\  + 


C((l  +  hL0)n-l)) 
L0 


where  C  =  max^  |/(0,  uo,  d)|. 

It’s  not  hard  to  show  Vffi)  has  the  following  property: 

Lemma  13:  V^(-)  G  V(M?),  and  the  coefficient  C  in  defining  P(M2)  can  be  made  independent 
of  h. 

Remark:  Lemma  13  implies  that  1 7^(0  is  locally  bounded  and  locally  Lipshitz  continuous. 

Following  standard  arguments,  one  can  show  Vffi)  satisfies  the  discrete  Bellman  equation 
(DBE): 

Theorem  14:  V/,  (•)  satisfies: 

(DBE)  Vh(x)  =  min{  inf  {(l—Xh)Vh(x+huf-(x))+hl(x,u)},  inf  {(l—Xh)Vk(x+huf+(x))+hl(x,u)}}. 

u£U _  U&U+ 

(49) 

It’s  of  interest  to  know  whether  (49)  characterizes  the  value  function  Vh(-).  Unlike  in  [2]  (Chapter 
VI),  where  a  bounded  value  function  was  considered,  we  have  14(-)  unbounded.  But  it  turns  out 
that  with  a  little  bit  additional  assumption,  (49)  has  a  unique  solution. 

Proposition  15:  Assume  (Ti)  and  (A3).  If  — <  1,  then  there  exists  unique  solution 
in  P(M2)  to  (49),  where 

Co  =  W\  ^  ■ 

Lf 

Proof.  Let  Vh(x)  =  I4(x)  <  x  >~m,m  >  2,  where  <  x  >=  (1  +  |x| 2)1//2.  Since  Vh  G  P(M2),  14  is 
bounded.  In  terms  of  V/,.,  (49)  is  rewritten  as 

Vh (x)  =  min{  inf  {(1  —  A h)Vh(x  +  hf-(x,u))(— — — — ^  ^  ’ — ^-^-)m  +  hl(x,  u)  <  x  >~m}, 
ueU-  <  x  > 

inf  {(1  -  Xh))Vh(x  +  hf+(x,  u))(K  X  +  fe/+(X’ U)  >  )m  +  hl(x,  u)<x  >~m}}  (50) 

u£U+  <  X  > 

=■■  (G(Vh))(x). 

It  suffices  to  show  (50)  has  a  unique  solution.  It’s  clear  that  the  operator  G(-)  maps  any  W  G 
BC( M2)  into  BC(R2),  where  BC(K.2)  denotes  the  set  of  bounded  continuous  functions.  Now  take 
V.  W  gRC*(M2).  For  any  x,  without  loss  of  generality,  assume  u\  G  U-  achieves  the  minimum  in 
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G(V)(x).  Then 


G(W)(x)  -  G(V)(x) 


<  (1  -  A h)(<x  +  hf-(x>Ul)  >')mw(x  +  hf-(x,  u)) 


<  x  > 

-(1  -  >t)(<^V-(»."l)>ry(l  +  ,,/_(„)) 
_  <x  +  h/.(x,u l)>  ^  _  C)(x). 

<  1  >  rf 


Using  the  boundness  of  /_ ,  one  can  show  that 

,<  a  +  fe/-(x,m)  >^m  <  ,  2 

<x>  -lV^+4-C0J 

Since  we  may  choose  m  arbitrarily  close  to  2, 

G(IV)(x)  -  G(U)(x)  <  p  sup  |  (IT  -  V)(x)|, 

xel2 


where  p  <  1.  By  symmetry,  we  have 

G(U)(x)  -  G(W)(x)  <  p  sup  | (W  -  V)(x)\. 

xeK2 

Thus  ||  G(W)  —  G(V)  ||oo<  p  ||  W  —  V  1 1 oo .  Since  BC(M2)  is  a  Banach  space,  by  Contraction 
Mapping  Principle,  there  exists  a  unique  solution  to  (50)  in  BC^M2).  □ 

The  following  theorem  asserts  that  Vh(-)  converges  to  U(-)  as  h  — >  0.  The  proof  can  be  found 
in  [2] (Chapter  VI) (with  minor  modification). 

Theorem  16[2]:  Under  assumptions  which  guarantee  uniqueness  of  (HJB)  and  (DBE), 

sup  \  Vh(x)  —  V(a)|  — >  0  as  h —*  0,  (51) 

x  s/C 

for  every  compact  1C C  M2,  where  V^(-)  and  V(-)  are  the  unique  solutions  to  (DBE)  and  (HJB) 
respectively. 

Remarks: 


•  Theorem  16  also  serves  as  a  proof  of  existence  of  a  solution  to  (HJB). 

•  In  solving  (49),  one  obtains  the  optimal  control  a£[-]  for  the  discrete  time  problem.  A  sub- 
optimal  control  for  the  continuous  time  problem  is  «/,.(•)  defined  by  a/,(i)  =  oJh[k],  Vt  G 
[kh,  (k  +  1  )h),  k  G  N.  As  h  — »  0,  one  can  show  J(x,  cth(-))  —■ >  V(x). 
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5.2  Fully-discrete  approximation 

Theoretically  the  solution  to  (49)  can  be  obtained  by  first  computingF(-)  via  successive  approxima¬ 
tion  and  then  transforming  back  to  F(-).  An  approximation  scheme  for  solving  (DBE)  is  described 
in  [2] (Appendix  A).  It  was  shown  there  that  when  space  discretization  gets  finer  and  finer,  the 
solution  obtained  via  solving  a  finite  system  of  equations  converges  to  V/,(-). 


6  Conclusions  and  Future  Work 

In  this  report,  we  have  studied  control  of  a  magnetostrictive  actuator,  taking  the  infinite  horizon 
optimal  control  problem  as  an  example.  We  characterized  the  value  function  as  the  (unique)  solution 
of  a  Hamilton- Jacob i-Bellman  equation  of  a  hybrid  form.  And  we  pointed  out  how  to  solve  the 
(HJB)  and  obtain  a  suboptimal  control  by  discrete  time  approximation. 

Future  work  includes  extension  of  this  approach  to  other  control  problems  of  practical  interests, 
which  are  listed  below: 

•  Finite  Horizon  Control  Problems.  Such  problems  arise,  for  instance,  in  tracking  control 
of  the  actuator. 

•  Problems  with  State-Space  Constraints.  The  state-space  constraints  come  from  physics, 
as  well  as  limitation  on  the  operating  range  of  the  input  current. 

•  Time-Optimal  Control  Problem.  An  important  example  of  this  is  fast  positioning. 

•  Control  Problem.  To  account  for  exogenous  disturbances  and  unmodeled  dynamics, 
we  can  introduce  a  noise  term  into  the  model  and  investigate  robust  control  of  the  actuator 
using  Hoc  control  theory. 

For  each  of  the  problems  discussed  above,  we  need  to  study  the  exact  form  of  the  Hamilton- 
Jacobi  equation  (together  with  its  initial/boundary  conditions)  satisfied  by  the  value  function,  the 
numerical  solution  of  the  equation  and  the  controller  synthesis  method. 


A  Proof  of  Lemma  1 

Proof. 
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9C(z) 

dz 


—  csch2(z ) 


(eZ~2~Z)2 


Z 2  (-+#  +  # 


Therefore 


dC(z) 

dz 


>  0,  Vz  /  0. 


Further  manipulation  on  (52)  yields 

dC(z)  _  (2  +  |r  +  |rH - )(^r  +  fr 


+  •••) 


dz 


from  which  we  obtain 


(l  +  §  +  ff  +  ---)5 


£<»>  ^  I- 


Combining  (53)  and  (55)  we  have 
Since  in  addition, 


dC(z) 

dz 


>  o. 


oo 


lim  £( z)  =  1,  lim  £(z) 

Z— XX)  “  ' 

we  have  (13). 

To  prove  it  suffices  to  show 

d2C(z) 


=  -1, 


dz2 
d2C(z' 
dz 2 


>  0  Mz  <  0, 

■  <  0  Vz  >  0. 


But 


d2C(z)  _  S(ez  +  e~z)  2_ 

(ez  -  e~z)3  z 3 


dz2 


=  2- 


(i  +  fr  +  frH —  )-(i  +  fr  +  fr  +  '-  -) 


^3(i  +  !r  +  ff  +  ---)2 


(52) 

(53) 


(55) 

(56) 

(57) 


(59) 


so  we  need  only  to  show  that  the  numerator  of  (59)  is  always  less  than  0  Vz  =/=■  0.  We  first  note  that 
the  coefficient  of  z2k,  k  >  1  in  the  second  term  is 


3( 


1 


+ 


1 


(2k  +  1)!  (2k  —  1)!3! 


+  •••)  >  3 


1 


> 


> 


(2k  —  1)!3! 

1  m 


(2  k)r  3! 

1 


(2k)!’ 
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while  ^yr  is  the  coefficient  of  z2k  in  the  first  term.  For  k  =  0, 1,  the  coefficients  of  both  terms 
cancel  out.  The  proof  is  now  complete.  □ 
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